Web Scraping in R

Denise Bradford

January 13, 2022

A web of data

  • In 2008, an estimated 154 million HTML tables (out of the 14.1 billion) contain ‘high quality relational data’!!!
  • Hard to quantify how much more exists outside of HTML Tables, but there is an estimate of at least 30 million lists with ‘high quality relational data’.
  • A growing number of websites/companies provide programmatic access to their data/services via web APIs (that data typically comes in XML/JSON format).

Before scraping, do some googling!

A web of messy data!

  • Recall the concept of tidy data.
  • Data is in a table where
    • 1 row == 1 observation
    • 1 column == 1 variable (observational attribute)
  • Parsing web data (HTML/XML/JSON) is easy (for computers)
  • Getting it in a tidy form is typically not easy.
  • Knowing a bit about modern tools & web technologies makes it much easier.

What is webscraping?

  • Extract data from websites
    • Tables
    • Links to other websites
    • Text

Why webscrape?

  • Because copy-paste is awful
  • Because it’s fast
  • Because you can automate it

Don’t abuse your power

  • Before scraping a website, please read the terms and conditions!!
  • It’s sometimes more efficient/appropriate to find the API.
  • If a website public offers an API, USE IT (instead of scraping)!!! (more on this later)

http://www.wired.com/2014/01/how-to-hack-okcupid

http://www.vox.com/2016/5/12/11666116/70000-okcupid-users-data-release

Webscraping with rvest:
Step-by-Step Start Guide

Step 1: Find a URL

What data do you want?

  • Information on Oscar-nominated winning film Moonlight

Find it on the web!

# character variable containing the url you want to scrape
myurl <- "http://www.imdb.com/title/tt4975722/"

Step 2: Read HTML into R

“Huh? What am I doing?” - some of you right now

  • HTML is HyperText Markup Language. All webpages are written with it.
  • Go to any website, right click, click “View Page Source” to see the HTML
library(tidyverse)
library(rvest)
myhtml <- read_html(myurl)
myhtml
## {html_document}
## <html lang="en-US" xmlns:og="http://opengraphprotocol.org/schema/" xmlns:fb="http://www.facebook.com/2008/fbml">
## [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
## [2] <body>\n<div>    <img height="1" width="1" style="display:none;visibility ...

Step 3: Figure out
where your data is

Need to find your data within the myhtml object.

Tags to look for:

  • <p>: paragraphs
  • <h1>, <h2>, etc.: headers
  • <a>: links
  • <li>: item in a list
  • <table>: tables

Use Selector Gadget to find the exact location. (Demo)

For more on HTML, I recommend W3schools’ tutorial

  • You don’t need to be an expert in HTML to web scrape with rvest!

Step 4: Tell rvest where to find your data

Copy-paste from Selector Gadget or give HTML tags into html_nodes() to extract your data of interest

#myhtml %>% html_nodes("plotsummary") #%>% html_text() 
#myhtml %>% html_nodes("table") %>% html_table(header = TRUE)

Step 5: Save & tidy data

library(stringr)
library(magrittr)
# mydat <- myhtml %>% 
#   html_nodes("table") %>%
#   magrittr::extract2(1) %>% 
#   html_table(header = TRUE)
# 
# mydat <- mydat[,c(2,4)]
# names(mydat) <- c("Actor", "Role")
# mydat <- mydat %>% 
#   mutate(Actor = Actor,
#          Role = str_replace_all(Role, "\n  ", ""))
# mydat

Your Turn #1

Using rvest, scrape a table from Wikipedia. You can pick your own table or you can get one of the tables in the country GDP per capita example from earlier.

Your result should be a data frame with one observation per row and one variable per column.

Your Turn #1 Solution

library(rvest)
library(magrittr)
myurl <- "https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP)_per_capita"
myhtml <- read_html(myurl)
myhtml %>% 
 html_nodes("table") %>%
 magrittr::extract2(2) %>%
 html_table(header = TRUE) %>% 
 #dplyr::mutate(`Int$` = parse_number(`Int$`)) %>% 
 head
## # A tibble: 6 × 9
##   `Country/Territory` Subregion         Region `IMF[5]` `IMF[5]` `World Bank[6]`
##   <chr>               <chr>             <chr>  <chr>    <chr>    <chr>          
## 1 Country/Territory   Subregion         Region Estimate Year     Estimate       
## 2 Liechtenstein *     Western Europe    Europe N/A      N/A      N/A            
## 3 Luxembourg *        Western Europe    Europe 122,740  2021     118,360        
## 4 Monaco *            Western Europe    Europe N/A      N/A      N/A            
## 5 Singapore *         South-eastern As… Asia   102,742  2021     98,526         
## 6 Ireland *           Northern Europe   Europe 99,239   2021     93,612         
## # … with 3 more variables: World Bank[6] <chr>, CIA[7] <chr>, CIA[7] <chr>

Deeper dive into rvest

Key Functions: html_nodes

  • html_nodes(x, "path") extracts all elements from the page x that have the tag / class / id path. (Use SelectorGadget to determine path.)
  • html_node() does the same thing but only returns the first matching element.
  • Can be chained
myhtml %>% 
  html_nodes("p") %>% # first get all the paragraphs 
  html_nodes("a") # then get all the links in those paragraphs
## {xml_nodeset (40)}
##  [1] <a href="/wiki/Gross_domestic_product" title="Gross domestic product">gr ...
##  [2] <a href="/wiki/Purchasing_power_parity" title="Purchasing power parity"> ...
##  [3] <a href="/wiki/Goods_and_services" title="Goods and services">goods and  ...
##  [4] <a href="/wiki/International_dollar" title="International dollar">Int$</a>
##  [5] <a href="#cite_note-world-2019-3">[n 1]</a>
##  [6] <a href="/wiki/List_of_countries_by_wealth_per_adult" title="List of cou ...
##  [7] <a href="/wiki/Gross_domestic_product" title="Gross domestic product">gr ...
##  [8] <a href="/wiki/Per_capita" title="Per capita">per capita</a>
##  [9] <a href="/wiki/IMF" class="mw-redirect" title="IMF">IMF</a>
## [10] <a href="/wiki/World_Bank" title="World Bank">World Bank</a>
## [11] <a href="/wiki/Savings" class="mw-redirect" title="Savings">savings</a>
## [12] <a href="/wiki/Cost_of_living" title="Cost of living">cost of living</a>
## [13] <a href="/wiki/List_of_countries_by_GDP_(nominal)_per_capita" title="Lis ...
## [14] <a href="https://en.wiktionary.org/wiki/generalized" class="extiw" title ...
## [15] <a href="/wiki/Living_standards" class="mw-redirect" title="Living stand ...
## [16] <a href="/wiki/Inflation_rates" class="mw-redirect" title="Inflation rat ...
## [17] <a href="/wiki/Exchange_rates" class="mw-redirect" title="Exchange rates ...
## [18] <a href="#cite_note-4">[3]</a>
## [19] <a href="#cite_note-5">[4]</a>
## [20] <a href="/wiki/Personal_income" title="Personal income">personal income</a>
## ...

Key Functions: html_text

  • html_text(x) extracts all text from the nodeset x
  • Good for cleaning output
myhtml %>% 
  html_nodes("p") %>% # first get all the paragraphs 
  html_nodes("a") %>% # then get all the links in those paragraphs
  html_text() # get the linked text only 
##  [1] "gross domestic product"                       
##  [2] "purchasing power parity"                      
##  [3] "goods and services"                           
##  [4] "Int$"                                         
##  [5] "[n 1]"                                        
##  [6] "list of countries by wealth per adult"        
##  [7] "gross domestic product"                       
##  [8] "per capita"                                   
##  [9] "IMF"                                          
## [10] "World Bank"                                   
## [11] "savings"                                      
## [12] "cost of living"                               
## [13] "List of countries by GDP (nominal) per capita"
## [14] "generalized"                                  
## [15] "living standards"                             
## [16] "inflation rates"                              
## [17] "exchange rates"                               
## [18] "[3]"                                          
## [19] "[4]"                                          
## [20] "personal income"                              
## [21] "Standard of living and GDP"                   
## [22] "international dollars"                        
## [23] "rounded"                                      
## [24] "whole number"                                 
## [25] "[8]"                                          
## [26] "[9]"                                          
## [27] "[10]"                                         
## [28] "[11]"                                         
## [29] "tax havens"                                   
## [30] "corporate tax havens"                         
## [31] "[12]"                                         
## [32] "tax haven lists"                              
## [33] "[13]"                                         
## [34] "[14]"                                         
## [35] "leprechaun economics"                         
## [36] "BEPS"                                         
## [37] "modified gross national income"               
## [38] "corporate tax havens"                         
## [39] "major global tax havens"                      
## [40] "GDP-per-capita tax haven proxy"

Key Functions: html_table

  • html_table(x, header, fill) - parse html table(s) from x into a data frame or list of data frames
  • Structure of HTML makes finding and extracting tables easy!
myhtml %>% 
  html_nodes("table") %>% # get the tables 
  head(2) # look at first 2
## {xml_nodeset (2)}
## [1] <table width="100%"><tbody><tr>\n<td valign="top"> <style data-mw-dedupli ...
## [2] <table class="wikitable sortable static-row-numbers plainrowheaders srn-w ...
myhtml %>% 
  html_nodes("table") %>% # get the tables 
  magrittr::extract2(2) %>% # pick the second one to parse
  html_table(header = TRUE) # parse table 
## # A tibble: 229 × 9
##    `Country/Territory` Subregion       Region  `IMF[5]` `IMF[5]` `World Bank[6]`
##    <chr>               <chr>           <chr>   <chr>    <chr>    <chr>          
##  1 Country/Territory   Subregion       Region  Estimate Year     Estimate       
##  2 Liechtenstein *     Western Europe  Europe  N/A      N/A      N/A            
##  3 Luxembourg *        Western Europe  Europe  122,740  2021     118,360        
##  4 Monaco *            Western Europe  Europe  N/A      N/A      N/A            
##  5 Singapore *         South-eastern … Asia    102,742  2021     98,526         
##  6 Ireland *           Northern Europe Europe  99,239   2021     93,612         
##  7 Qatar *             Western Asia    Asia    97,262   2021     89,949         
##  8 Macau *             Eastern Asia    Asia    90,606   2021     57,807         
##  9 Isle of Man *       Northern Europe Europe  N/A      N/A      N/A            
## 10 Bermuda *           Northern Ameri… Americ… N/A      N/A      85,264         
## # … with 219 more rows, and 3 more variables: World Bank[6] <chr>,
## #   CIA[7] <chr>, CIA[7] <chr>

Key functions: html_attrs

  • html_attrs(x) - extracts all attribute elements from a nodeset x
  • html_attr(x, name) - extracts the name attribute from all elements in nodeset x
  • Attributes are things in the HTML like href, title, class, style, etc.
  • Use these functions to find and extract your data
myhtml %>% 
  html_nodes("table") %>% 
  magrittr::extract2(2) %>%
  html_attrs()
##                                                                        class 
## "wikitable sortable static-row-numbers plainrowheaders srn-white-background" 
##                                                                       border 
##                                                                          "1" 
##                                                                        style 
##                                                          "text-align:right;"
myhtml %>% 
  html_nodes("p") %>% 
  html_nodes("a") %>%
  html_attr("href")
##  [1] "/wiki/Gross_domestic_product"                                                                  
##  [2] "/wiki/Purchasing_power_parity"                                                                 
##  [3] "/wiki/Goods_and_services"                                                                      
##  [4] "/wiki/International_dollar"                                                                    
##  [5] "#cite_note-world-2019-3"                                                                       
##  [6] "/wiki/List_of_countries_by_wealth_per_adult"                                                   
##  [7] "/wiki/Gross_domestic_product"                                                                  
##  [8] "/wiki/Per_capita"                                                                              
##  [9] "/wiki/IMF"                                                                                     
## [10] "/wiki/World_Bank"                                                                              
## [11] "/wiki/Savings"                                                                                 
## [12] "/wiki/Cost_of_living"                                                                          
## [13] "/wiki/List_of_countries_by_GDP_(nominal)_per_capita"                                           
## [14] "https://en.wiktionary.org/wiki/generalized"                                                    
## [15] "/wiki/Living_standards"                                                                        
## [16] "/wiki/Inflation_rates"                                                                         
## [17] "/wiki/Exchange_rates"                                                                          
## [18] "#cite_note-4"                                                                                  
## [19] "#cite_note-5"                                                                                  
## [20] "/wiki/Personal_income"                                                                         
## [21] "/wiki/Gross_domestic_product#Standard_of_living_and_GDP:_wealth_distribution_and_externalities"
## [22] "/wiki/International_dollar"                                                                    
## [23] "/wiki/Rounding"                                                                                
## [24] "/wiki/Integer"                                                                                 
## [25] "#cite_note-9"                                                                                  
## [26] "#cite_note-10"                                                                                 
## [27] "#cite_note-11"                                                                                 
## [28] "#cite_note-12"                                                                                 
## [29] "/wiki/Tax_havens"                                                                              
## [30] "/wiki/Corporate_tax_haven"                                                                     
## [31] "#cite_note-qqtz-13"                                                                            
## [32] "/wiki/Tax_haven#Tax_haven_lists"                                                               
## [33] "#cite_note-dhar-14"                                                                            
## [34] "#cite_note-imfx-15"                                                                            
## [35] "/wiki/Leprechaun_economics"                                                                    
## [36] "/wiki/BEPS"                                                                                    
## [37] "/wiki/Modified_gross_national_income"                                                          
## [38] "/wiki/Corporate_tax_haven"                                                                     
## [39] "/wiki/Tax_haven#Top_10_tax_havens"                                                             
## [40] "/wiki/Corporate_haven#GDP-per-capita_tax_haven_proxy"

Other functions

  • html_children - list the “children” of the HTML page. Can be chained like html_nodes
  • html_name - gives the tags of a nodeset. Use in a chain with html_children
myhtml %>% 
  html_children() %>% 
  html_name()
## [1] "head" "body"
  • html_form - parses HTML forms (checkboxes, fill-in-the-blanks, etc.)
  • html_session - simulate a session in an html browser; use the functions jump_to, back to navigate through the page

Your Turn #2

Find another website you want to scrape (ideas: all bills in the house so far this year, video game reviews, anything Wikipedia) and use at least 3 different rvest functions in a chain to extract some data.

Advanced Example: Inaugural Addresses

The Data

  • The Avalon Project has most of the U.S. Presidential inaugural addresses.
  • Obama & Trump’s (’13, ’17), VanBuren 1837, Buchanan 1857, Garfield 1881, and Coolidge 1925 are missing, but are easily found elsewhere. I have them saved as text files on the website.
  • Let’s scrape all of them from The Avalon Project!

Get data frame of addresses

  • Could use another source to get this data of President names and years of inaugurations, but we’ll use The Avalon Project’s site because it’s a good example of data that needs tidying.
url <- "http://avalon.law.yale.edu/subject_menus/inaug.asp"
# even though it's called "all inaugs" some are missing
all_inaugs <- (url %>% 
  read_html() %>% 
  html_nodes("table") %>% 
  html_table(fill=T, header = T)) %>% extract2(3)
# tidy table of addresses
all_inaugs_tidy <- all_inaugs %>% 
  gather(term, year, -President) %>% 
  filter(!is.na(year)) %>% 
  select(-term) %>% 
  arrange(year)
head(all_inaugs_tidy)
## # A tibble: 6 × 2
##   President          year
##   <chr>             <int>
## 1 George Washington  1789
## 2 George Washington  1793
## 3 John Adams         1797
## 4 Thomas Jefferson   1801
## 5 Thomas Jefferson   1805
## 6 James Madison      1809

Automate scraping

  • A function to read the addresses and get the text of the speeches, with a catch for a read error
get_inaugurations <- function(url){
  test <- try(url %>% read_html(), silent=T)
  if ("try-error" %in% class(test)) {
    return(NA)
  } else
    url %>% read_html() %>%
      html_nodes("p") %>% 
      html_text() -> address
    return(unlist(address))
}

# takes about 30 secs to run
all_inaugs_text <- all_inaugs_tidy %>% 
  mutate(address_text = (map(url, get_inaugurations))) 

all_inaugs_text$address_text[[1]]
## [1] " Fellow-Citizens of the Senate and of the House of Representatives: "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
## [2] "Among the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the 14th day of the present month. On the one hand, I was summoned by my Country, whose voice I can never hear but with veneration and love, from a retreat which I had chosen with the fondest predilection, and, in my flattering hopes, with an immutable decision, as the asylum of my declining years--a retreat which was rendered every day more necessary as well as more dear to me by the addition of habit to inclination, and of frequent interruptions in my health to the gradual waste committed on it by time. On the other hand, the magnitude and difficulty of the trust to which the voice of my country called me, being sufficient to awaken in the wisest and most experienced of her citizens a distrustful scrutiny into his qualifications, could not but overwhelm with despondence one who (inheriting inferior endowments from nature and unpracticed in the duties of civil administration) ought to be peculiarly conscious of his own deficiencies. In this conflict of emotions all I dare aver is that it has been my faithful study to collect my duty from a just appreciation of every circumstance by which it might be affected. All I dare hope is that if, in executing this task, I have been too much swayed by a grateful remembrance of former instances, or by an affectionate sensibility to this transcendent proof of the confidence of my fellow-citizens, and have thence too little consulted my incapacity as well as disinclination for the weighty and untried cares before me, my error will be palliated by the motives which mislead me, and its consequences be judged by my country with some share of the partiality in which they originated. "                                                                                                                                                                                                                                                                                                                                                                                           
## [3] "Such being the impressions under which I have, in obedience to the public summons, repaired to the present station, it would be peculiarly improper to omit in this first official act my fervent supplications to that Almighty Being who rules over the universe, who presides in the councils of nations, and whose providential aids can supply every human defect, that His benediction may consecrate to the liberties and happiness of the people of the United States a Government instituted by themselves for these essential purposes, and may enable every instrument employed in its administration to execute with success the functions allotted to his charge. In tendering this homage to the Great Author of every public and private good, I assure myself that it expresses your sentiments not less than my own, nor those of my fellow- citizens at large less than either. No people can be bound to acknowledge and adore the Invisible Hand which conducts the affairs of men more than those of the United States. Every step by which they have advanced to the character of an independent nation seems to have been distinguished by some token of providential agency; and in the important revolution just accomplished in the system of their united government the tranquil deliberations and voluntary consent of so many distinct communities from which the event has resulted can not be compared with the means by which most governments have been established without some return of pious gratitude, along with an humble anticipation of the future blessings which the past seem to presage. These reflections, arising out of the present crisis, have forced themselves too strongly on my mind to be suppressed. You will join with me, I trust, in thinking that there are none under the influence of which the proceedings of a new and free government can more auspiciously commence. "                                                                                                                                                                                                                                                                                                                                              
## [4] "By the article establishing the executive department it is made the duty of the President \"to recommend to your consideration such measures as he shall judge necessary and expedient.\" The circumstances under which I now meet you will acquit me from entering into that subject further than to refer to the great constitutional charter under which you are assembled, and which, in defining your powers, designates the objects to which your attention is to be given. It will be more consistent with those circumstances, and far more congenial with the feelings which actuate me, to substitute, in place of a recommendation of particular measures, the tribute that is due to the talents, the rectitude, and the patriotism which adorn the characters selected to devise and adopt them. In these honorable qualifications I behold the surest pledges that as on one side no local prejudices or attachments, no separate views nor party animosities, will misdirect the comprehensive and equal eye which ought to watch over this great assemblage of communities and interests, so, on another, that the foundation of our national policy will be laid in the pure and immutable principles of private morality, and the preeminence of free government be exemplified by all the attributes which can win the affections of its citizens and command the respect of the world. I dwell on this prospect with every satisfaction which an ardent love for my country can inspire, since there is no truth more thoroughly established than that there exists in the economy and course of nature an indissoluble union between virtue and happiness; between duty and advantage; between the genuine maxims of an honest and magnanimous policy and the solid rewards of public prosperity and felicity; since we ought to be no less persuaded that the propitious smiles of Heaven can never be expected on a nation that disregards the eternal rules of order and right which Heaven itself has ordained; and since the preservation of the sacred fire of liberty and the destiny of the republican model of government are justly considered, perhaps, as deeply, as finally, staked on the experiment entrusted to the hands of the American people. "
## [5] "Besides the ordinary objects submitted to your care, it will remain with your judgment to decide how far an exercise of the occasional power delegated by the fifth article of the Constitution is rendered expedient at the present juncture by the nature of objections which have been urged against the system, or by the degree of inquietude which has given birth to them. Instead of undertaking particular recommendations on this subject, in which I could be guided by no lights derived from official opportunities, I shall again give way to my entire confidence in your discernment and pursuit of the public good; for I assure myself that whilst you carefully avoid every alteration which might endanger the benefits of an united and effective government, or which ought to await the future lessons of experience, a reverence for the characteristic rights of freemen and a regard for the public harmony will sufficiently influence your deliberations on the question how far the former can be impregnably fortified or the latter be safely and advantageously promoted. "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
## [6] "To the foregoing observations I have one to add, which will be most properly addressed to the House of Representatives. It concerns myself, and will therefore be as brief as possible. When I was first honored with a call into the service of my country, then on the eve of an arduous struggle for its liberties, the light in which I contemplated my duty required that I should renounce every pecuniary compensation. From this resolution I have in no instance departed; and being still under the impressions which produced it, I must decline as inapplicable to myself any share in the personal emoluments which may be indispensably included in a permanent provision for the executive department, and must accordingly pray that the pecuniary estimates for the station in which I am placed may during my continuance in it be limited to such actual expenditures as the public good may be thought to require. "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
## [7] "Having thus imparted to you my sentiments as they have been awakened by the occasion which brings us together, I shall take my present leave; but not without resorting once more to the benign Parent of the Human Race in humble supplication that, since He has been pleased to favor the American people with opportunities for deliberating in perfect tranquillity, and dispositions for deciding with unparalleled unanimity on a form of government for the security of their union and the advancement of their happiness, so His divine blessing may be equally conspicuous in the enlarged views, the temperate consultations, and the wise measures on which the success of this Government must depend. "

Add Missings

all_inaugs_text$President[is.na(all_inaugs_text$address_text)]
## [1] "Martin Van Buren"  "James Buchanan"    "James A. Garfield"
## [4] "Calvin Coolidge"
# there are 7 missing at this point: obama's and trump's, plus coolidge, garfield, buchanan, and van buren, which errored in the scraping.
obama09 <- get_inaugurations("http://avalon.law.yale.edu/21st_century/obama.asp")
obama13 <- readLines("speeches/obama2013.txt")
trump17 <- readLines("speeches/trumpinaug.txt")
vanburen1837 <- readLines("speeches/vanburen1837.txt") # row 13
buchanan1857 <- readLines("speeches/buchanan1857.txt") # row 18
garfield1881 <- readLines("speeches/garfield1881.txt") # row 24
coolidge1925 <- readLines("speeches/coolidge1925.txt") # row 35
all_inaugs_text$address_text[c(13,18,24,35)] <- list(vanburen1837,buchanan1857, garfield1881, coolidge1925)

# lets combine them all now
recents <- data.frame(President = c(rep("Barack Obama", 2), 
                                    "Donald Trump"),
                      year = c(2009, 2013, 2017), 
                      url = NA,
                      address_text = NA)

all_inaugs_text <- rbind(all_inaugs_text, recents)
all_inaugs_text$address_text[c(56:58)] <- list(obama09, obama13, trump17)

Check-in: What did we do?

  1. We found some interesting data to scrape from the web using rvest.
  2. We used tidyr to create tidy data: A data frame of President and year. One observation per row!
  3. We used the consistent HTML structure of the urls we wanted to scrape to automate collection of web data
    • Way faster than copy-paste!
    • Though we had to do some by hand, we took advantage of the tidy data and added the missing data manually without much pain.
  4. We now have a tidy data set of Presidential inaugural addresses for text analysis!

A (Small) Text Analysis

Now, I use the tidytext package to get the words out of each inaugural address.

# install.packages("tidytext")
library(tidytext)
all_inaugs_text %>% 
  select(-url) %>% 
  unnest() %>% 
  unnest_tokens(word, address_text) -> presidential_words
## Warning: `cols` is now required when using unnest().
## Please use `cols = c(address_text)`
head(presidential_words)
## # A tibble: 6 × 3
##   President          year word    
##   <chr>             <dbl> <chr>   
## 1 George Washington  1789 fellow  
## 2 George Washington  1789 citizens
## 3 George Washington  1789 of      
## 4 George Washington  1789 the     
## 5 George Washington  1789 senate  
## 6 George Washington  1789 and

Longest speeches

presidential_words %>% 
  group_by(President,year) %>% 
  summarize(num_words = n()) %>%
  arrange(desc(num_words)) -> presidential_wordtotals
## `summarise()` has grouped output by 'President'. You can override using the `.groups` argument.
## Warning: ggrepel: 50 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Web APIs

Web APIs

  • Server-side Web APIs (Application Programming Interfaces) are a popular way to provide easy access to data and other services.
  • If you (the client) want data from a server, you typically need one HTTP verb – GET.
library(httr)
sam <- GET("https://api.github.com/users/sctyner")
content(sam)[c("name", "company")]
## $name
## [1] "Sam Tyner"
## 
## $company
## [1] "@Tritura"
  • Other HTTP verbs – POST, PUT, DELETE, etc…

Request/response model

  • When you (the client) requests a resource from the server. The server responds with a bunch of additional information.
sam$header[1:3]
## $server
## [1] "GitHub.com"
## 
## $date
## [1] "Wed, 05 Jan 2022 18:24:36 GMT"
## 
## $`content-type`
## [1] "application/json; charset=utf-8"
  • Nowadays content-type is usually XML or JSON (HTML is great for sharing content between people, but it isn’t great for exchanging data between machines.)

Non-HTML Data Formats

What is XML?

XML is a markup language that looks very similar to HTML.

<mariokart>
  <driver name="Bowser" occupation="Koopa">
    <vehicle speed="55" weight="25"> Wario Bike </vehicle>
    <vehicle speed="40" weight="67"> Piranha Prowler </vehicle>
  </driver>
  <driver name="Peach" occupation="Princess">
    <vehicle speed="54" weight="29"> Royal Racer </vehicle>
    <vehicle speed="50" weight="34"> Wild Wing </vehicle>
  </driver>
</mariokart>
  • This example shows that XML can (and is) used to store inherently tabular data (thanks Jeroen Ooms)
  • What is are the observational units here? How many observations in total?
  • 2 units and 6 total observations (4 vehicles and 2 drivers).

XML2R

XML2R is a framework to simplify acquistion of tabular/relational XML.

## # A tibble: 6 × 1
##   obs          
##   <named list> 
## 1 <chr [1 × 4]>
## 2 <chr [1 × 4]>
## 3 <chr [1 × 3]>
## 4 <chr [1 × 4]>
## 5 <chr [1 × 4]>
## 6 <chr [1 × 3]>
## 
##          mariokart//driver mariokart//driver//vehicle 
##                          2                          4
  • XML2R coerces XML into a flat list of observations.
  • The list names track the “observational unit”.
  • The list values track the “observational attributes”.
obs # named list of observations
## $`mariokart//driver//vehicle`
##      speed weight XML_value     
## [1,] "55"  "25"   " Wario Bike "
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## 
## $`mariokart//driver//vehicle`
##      speed weight XML_value          
## [1,] "40"  "67"   " Piranha Prowler "
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## 
## $`mariokart//driver`
##      name     occupation
## [1,] "Bowser" "Koopa"   
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## 
## $`mariokart//driver//vehicle`
##      speed weight XML_value      
## [1,] "54"  "29"   " Royal Racer "
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## 
## $`mariokart//driver//vehicle`
##      speed weight XML_value    
## [1,] "50"  "34"   " Wild Wing "
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## 
## $`mariokart//driver`
##      name    occupation
## [1,] "Peach" "Princess"
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
collapse_obs(obs) # group into table(s) by observational name/unit
## $`mariokart//driver`
##      name     occupation
## [1,] "Bowser" "Koopa"   
## [2,] "Peach"  "Princess"
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [2,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## 
## $`mariokart//driver//vehicle`
##      speed weight XML_value          
## [1,] "55"  "25"   " Wario Bike "     
## [2,] "40"  "67"   " Piranha Prowler "
## [3,] "54"  "29"   " Royal Racer "    
## [4,] "50"  "34"   " Wild Wing "      
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [2,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [3,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [4,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
  • What information have I lost?
  • I can’t map vehicles to the drivers!
library(dplyr)
obs <- add_key(obs, parent = "mariokart//driver", recycle = "name")
## A key for the following children will be generated for the mariokart//driver node:
## mariokart//driver//vehicle
collapse_obs(obs)
## $`mariokart//driver`
##      name     occupation
## [1,] "Bowser" "Koopa"   
## [2,] "Peach"  "Princess"
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [2,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## 
## $`mariokart//driver//vehicle`
##      speed weight XML_value          
## [1,] "55"  "25"   " Wario Bike "     
## [2,] "40"  "67"   " Piranha Prowler "
## [3,] "54"  "29"   " Royal Racer "    
## [4,] "50"  "34"   " Wild Wing "      
##      url                                                                                                                           
## [1,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [2,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [3,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
## [4,] "https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml"
##      name    
## [1,] "Bowser"
## [2,] "Bowser"
## [3,] "Peach" 
## [4,] "Peach"

Now (if I want) I can merge the tables into a single table…

tabs <- collapse_obs(obs)
left_join(as.data.frame(tabs[[1]]), as.data.frame(tabs[[2]])) 
## Joining, by = c("name", "url")
##     name occupation
## 1 Bowser      Koopa
## 2 Bowser      Koopa
## 3  Peach   Princess
## 4  Peach   Princess
##                                                                                                                            url
## 1 https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml
## 2 https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml
## 3 https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml
## 4 https://gist.githubusercontent.com/cpsievert/85e340814cb855a60dc4/raw/651b7626e34751c7485cff2d7ea3ea66413609b8/mariokart.xml
##   speed weight         XML_value
## 1    55     25       Wario Bike 
## 2    40     67  Piranha Prowler 
## 3    54     29      Royal Racer 
## 4    50     34        Wild Wing

What about JSON?

  • JSON is the format for data on the web.
  • JavaScript Object Notation (JSON) is comprised of two components:
    • arrays => [value1, value2]
    • objects => {“key1”: value1, “key2”: [value2, value3]}
  • jsonlite is the preferred R package for parsing JSON (it’s even used by Shiny!)

Back to Mariokart

[
    {
        "driver": "Bowser",
        "occupation": "Koopa",
        "vehicles": [
            {
                "model": "Wario Bike",
                "speed": 55,
                "weight": 25
            },
            {
                "model": "Piranha Prowler",
                "speed": 40,
                "weight": 67
            }
        ]
    },
    {
        "driver": "Peach",
        "occupation": "Princess",
        "vehicles": [
            {
                "model": "Royal Racer",
                "speed": 54,
                "weight": 29
            },
            {
                "model": "Wild Wing",
                "speed": 50,
                "weight": 34
            }
        ]
    }
]
library(jsonlite)
## 
## Attaching package: 'jsonlite'
## The following object is masked from 'package:purrr':
## 
##     flatten
mario <- fromJSON("http://bit.ly/mario-json")
str(mario) 
## 'data.frame':    2 obs. of  3 variables:
##  $ driver    : chr  "Bowser" "Peach"
##  $ occupation: chr  "Koopa" "Princess"
##  $ vehicles  :List of 2
##   ..$ :'data.frame': 2 obs. of  3 variables:
##   .. ..$ model : chr  "Wario Bike" "Piranha Prowler"
##   .. ..$ speed : int  55 40
##   .. ..$ weight: int  25 67
##   ..$ :'data.frame': 2 obs. of  3 variables:
##   .. ..$ model : chr  "Royal Racer" "Wild Wing"
##   .. ..$ speed : int  54 50
##   .. ..$ weight: int  29 34
mario$driver
## [1] "Bowser" "Peach"
mario$vehicles
## [[1]]
##             model speed weight
## 1      Wario Bike    55     25
## 2 Piranha Prowler    40     67
## 
## [[2]]
##         model speed weight
## 1 Royal Racer    54     29
## 2   Wild Wing    50     34

How do we get two tables (with a common id) like the XML example?

vehicles <- rbind(mario$vehicles[[1]], mario$vehicles[[2]])
vehicles <- cbind(driver = mario$driver, vehicles)

Your Turn

  1. Get the json data for our R workshop GitHub commit history:
workshop_commits_raw <- fromJSON("https://api.github.com/repos/heike/rwrks/commits")

  1. Find the table of commits contained in this list. Hint: It’s all about the $

  2. Plot the total number of commits (number of rows) by user as a bar chart